DZone Spotlight

Agents are proliferating like wildfire, yet there is a ton of confusion surrounding foundational concepts such as agent observability. Is it the same as AI observability? What problem does it solve, and how does it work? Fear not, we'll dive into these questions and more. Along the way, we will cite specific user examples as well as our own experience in pushing a customer-facing AI agent into production. By the end of this article, you will understand: Best practices from real data + AI teamsHow the agent observability category is definedThe benefits of agent observabilityThe critical capabilities required for achieving those benefitsBest practices from real data + AI teams What Is an Agent? Anthropic defines an agent as “LLMs autonomously using tools in a loop.” I’ll expand on that definition a bit. An agent is an AI equipped with a set of guiding principles and resources, capable of a multi-step decision and action chain to produce a desired outcome. These resources often consist of access to databases, communication tools, or even other sub-agents (if you are using a multi-agent architecture). What is an agent? A visual guide to the agent lifecycle. Image courtesy of the author. For example, a customer support agent may: Receive a user inquiry regarding a refund on their last purchaseCreate and escalate a ticketAccess the relevant transaction history in the data warehouseAccess the relevant refund policy chunk in a vector databaseUse the provided context and instructional prompt to formulate a responseReply to the user And that would just be step one in the process! The user would reply creating another unique response and series of actions. What Is Observability? Observability is the ability to have visibility into a system's inputs and outputs, as well as the performance of its component parts. An analogy I like to use is a factory that produces widgets. You can test the widgets to make sure they are within spec, but to understand why any deficiencies occurred, you also need to monitor the gears that make up the assembly line (and have a process for fixing broken parts). The broken boxes represent data products, and the gears are the components in a data landscape that introduce reliability issues (data, systems, code). Image courtesy of the author. There are multiple observability categories. The term was first introduced by platforms designed to help software engineers or site reliability engineers reduce the time their applications are offline. These solutions are categorized by Gartner in their Magic Quadrant for Observability Platforms. Barr Moses introduced the data observability category in 2019. These platforms are designed to reduce data downtime and increase adoption of reliable data and AI. Gartner has produced a Data Observability Market Guide and given the category a benefit rating of HIGH. Gartner also projects 70% of organizations will adopt data observability platforms by 2027, an increase from 50% in 2025. And amidst these categories, you also have agent observability. Let’s define it. What Is Agent Observability? If we combine the two definitions — what is an agent and what is observability — together, we get the following: Agent observability is the ability to have visibility into the performance of the inputs, outputs, and component parts of an LLM system that uses tools in a loop. It’s a critical, fast-growing category — Gartner projects that 90% of companies with LLMs in production will adopt these solutions. Agent observability provides visibility into the agent lifecycle. Image courtesy of the author. Let’s revisit our customer success agent example to further flesh out this definition. What was previously an opaque process with a user question, “Can I get a refund?” and agent response, “Yes, you are within the 30-day return window. Would you like me to email you a return label?” now might look like this: Sample trace visualized. Image courtesy of the author. The above image is a visualized trace, or a record of each span (unit of work) the agent took as part of its session with a user. Many of these spans involve LLM calls. As you can see in the image below, agent observability provides visibility into the telemetry of each span, including the prompt (input), completion (output), and operational metrics such as token count (cost), latency, and more. As valuable as this visibility is, what is even more valuable is the ability to set proactive monitors on this telemetry. For example, getting alerted when the relevance of the agent output drops or if the amount of tokens used during a specific span starts to spike. We’ll dive into more details on common features, how it works, and best practices in subsequent sections, but first, let’s make sure we understand the benefits and goals of agent observability. A Quick Note on Synonymous Categories Terms like GenAI observability, AI observability, or LLM observability are often used interchangeably, although technically, the LLM is just one component of an agent. RAG (retrieval-augmented generation) observability refers to a similar but less narrow pattern involving AI retrieving context to inform its response. I’ve also seen teams reference LLMops, AgentOps, or evaluation platforms. The labels and technologies have evolved rapidly over a short period of time, but these categorical terms can be considered roughly synonymous. For example, Gartner has produced an “Innovation Insight: LLM Observability” report with essentially the same definition. Honestly, there is no need to sweat the semantics. Whatever you or your team decide to call it, what’s truly important is that you have the technology and processes in place to monitor and improve the quality and reliability of your agent’s outputs. Do You Need Agent Observability If You Use Guardrails? The short answer is yes. Many AI development platforms, such as AWS Bedrock, include real-time safeguards, called guardrails, to prevent toxic responses. However, guardrails aren’t designed to catch regressions in agent responses over time across dimensions such as accuracy, helpfulness, or relevance. In practice, you need both working together. Guardrails protect you from acute risks in real time, while observability protects you from chronic risks that appear gradually. It’s similar to the relationship between data testing and anomaly detection for monitoring data quality. Problem to Be Solved and Business Benefits Ultimately, the goal of any observability solution is to reduce and minimize downtime. This concept for software applications was popularized by the Google Site Reliability Engineering Handbook, which defined downtime as the portion of unsuccessful requests divided by the total number of requests. Like everything in the AI space, defining a successful request is more difficult than it seems. After all these are non-deterministic systems meaning you can provide the same input many times and get many different outputs. Is a request only unsuccessful if it technically fails? What about if it hallucinates and provides inaccurate information? What if the information is technically correct, but it’s in another language or surrounded by toxic language? Again, it’s best to avoid getting lost in the semantics and pedantics. Ultimately, the goal of reducing downtime is to ensure features are adopted and provide the intended value to users. This means agent downtime should be measured based on the underlying use case. For example, clarity and tone of voice might be paramount for our customer success chatbot, but it might not be a large factor for a revenue operations agent providing summarized insights from sales calls. This also means your downtime metric should correspond to user adoption. If those numbers don’t track, you haven’t captured the key metrics that make your agent valuable. Most data + AI teams I talk to today are using adoption as the main proxy for agent reliability. As the space begins to mature, teams are gradually moving toward more forward leading indicators such as downtime and the metrics that roll up to it such as relevancy, latency, recall (F1), and more. Dropbox, for example, measures agent downtime as: Responses without a citationIf more than 95% of responses have a latency greater than 5 secondsIf the agent does not reference the right source at least 85% of the time (F1 > 85%)Factual accuracy, clarity, and formatting are other dimensions, but a failure threshold isn’t provided. At Monte Carlo, our development team considers our Troubleshooting Agent as experiencing downtime based on the metrics of semantic distance, groundedness, and proper tool usage. These are evaluated on a 0-1 scale using an LLM-as-judge methodology. Downtime in staging is defined as: Any score under 0.5More than 33% of LLM-as-judge evaluations or more than 2 total evaluations score between a .5 and .8, even after an automatic retry. Groundedness tests show the agent invents information or answers out of scope (hallucination or missing context).The agent misuses or fails to use the required tools Outside of adoption, agents can be evaluated across the classic business values of reducing cost, increasing revenue, or decreasing risk. In these scenarios, the cost of downtime can be quantified easily by taking the frequency and duration of downtime and multiplying them by the ROI being driven by the agent. This formula remains mostly academic at the moment since, as we’ve noted previously, most teams are not as focused on measuring immediate ROI. However, I have spoken to a few. One of the clearest examples in this regard is a pharmaceutical company using an agent to enrich customer records in a master data management match-merge process. They originally built their business case on reducing cost, specifically the number of records that need to be enriched by human stewards. However, while they did increase the number of records that could be automatically enriched, they also improved a large number of poor records that would have been automatically discarded as well! So the human steward workload actually increased! Ultimately, this was a good result as record quality improved; however, it does underscore how fluid and unpredictable this space remains. How Agent Observability Works Agent observability can be built internally by engineering teams or purchased from several vendors. We’ll save the build vs. buy analysis for another time, but, as with data testing, some smaller teams will choose to start with an internal build until they reach a scale where a more systemic approach is required. Whether an internal build or vendor platform, when you boil it down to the essentials, there are really two core components to an agent observability platform: trace visualization and evaluation monitors. Trace Visualization Traces, or telemetry data that describes each step taken by an agent, can be captured using an open-source SDK that leverages the OpenTelemetry (Otel) framework. Teams label key steps — such as skills, workflows, or tool calls — as spans. When a session starts, the agent calls the SDK, which captures all the associated telemetry for each span, such as model version, duration, tokens, etc. A collector then sends that data to the intended destination (we think the best practice is to consolidate within your warehouse or lakehouse source of truth), where an application can then help visualize the information, making it easier to explore. One benefit to observing agent architectures is that this telemetry is relatively consolidated and easy to access via LLM orchestration frameworks, as compared to observing data architectures, where critical metadata may be spread across a half dozen systems. Evaluation Monitors Once you have all of this rich telemetry in place, you can monitor or evaluate it. This can be done using an agent observability platform, or sometimes the native capabilities within data + AI platforms. Teams will typically refer to the process of using AI to monitor AI (LLM-as-judge) as an evaluation. This type of monitor is well-suited to evaluate the helpfulness, validity, and accuracy of the agent. This is because the outputs are typically larger text fields and non-deterministic, making traditional SQL-based monitors less effective across these dimensions. Where SQL code-based monitors really shine, however, is in detecting issues across operational metrics (system failures, latency, cost, throughput) as well as situations in which the agent’s output must conform to a very specific format or rule. For example, if the output must be in the format of a US postal address, or if it must always have a citation. Most teams will require both types of monitors. In cases where either approach will produce a valid result, teams should favor code-based monitors as they are more deterministic, explainable, and cost-effective. However, it’s important to ensure your heuristic or code-based monitor is achieving the intended result. Simple code-based monitors focused on use case-specific criteria — say, output length must be under 350 characters–are typically more effective than complex formulas designed to broadly capture semantic accuracy or validity, such as ROUGE, BLEU, cosine similarity, and others. While these traditional metrics benefit from being explainable, they struggle when the same idea is expressed in different terms. Almost every data science team starts with these familiar monitors, only to quickly abandon them after a rash of false positives. What About Context Engineering and Reference Data? This is arguably the third component of agent observability. It can be a bit tricky to draw a firm line between data observability and agent observability — it's probably best not to even try. This is because agent behavior is driven by the data it retrieves, summarizes, or reasons over. In many cases, the “inputs” that shape an agent’s responses — things like vector embeddings, retrieval pipelines, and structured lookup tables — sit somewhere between the two worlds. Or perhaps it may be more accurate to say they all live in one world, and that agent observability MUST include data observability. This argument is pretty sound. After all, an agent can’t get the right answer if it’s fed wrong or incomplete context — and in these scenarios, agent observability evaluations will still pass with flying colors. Challenges and Best Practices It would be easy enough to generate a list of agent observability challenges teams could struggle with, but let’s take a look at the most common problems teams are actually encountering. And remember, these are challenges specifically related to observing agents. Challenge #1: Evaluation Cost LLM workloads aren’t cheap, and a single agent session can involve hundreds of LLM calls. Now imagine for each of those calls you are also calling another LLM multiple times to judge different quality dimensions. It can add up quickly. One data + AI leader confessed to us that their evaluation cost was 10 times as expensive as the baseline agent workload. Monte Carlo’s agent development team strives to maintain roughly a one to one workload to evaluation ratio. Best Practices to Contain Evaluation Cost Most teams will sample a percentage or an aggregate number of spans per trace to manage costs while still retaining the ability to detect performance degradations. Stratified sampling, or sampling a representative portion of the data, can be helpful in this regard. Conversely, it can also be helpful to filter for specific spans, such as those with a longer-than-average duration. Challenge #2: Defining Failure and Alert Conditions Even when teams have all the right telemetry and evaluation infrastructure in place, deciding what actually constitutes “failure” can be surprisingly difficult. To start, defining failure requires a deep understanding of the agent’s use case and user expectations. A customer support bot, a sales assistant, and a research summarizer all have different standards for what counts as “good enough.” What’s more, the relationship between a bad response and its real-world impact on adoption isn’t always linear or obvious. For example, if an evaluation model gives a response that is judged to be a .75 for clarity, is that a failure? Best Practices for Defining Failure and Alert Conditions Aggregate multiple evaluation dimensions. Rather than declaring a failure based on a single score, combine several key metrics — such as helpfulness, accuracy, faithfulness, and clarity — and treat them as a composite pass/fail test. This is the approach Monte Carlo takes in our agent evaluation framework for our internal agents. Most teams will also leverage anomaly detection to identify a consistent drop in scores over a period of time rather than a single (possibly hallucinated) evaluation. Dropbox, for example, leverages dashboards that track their evaluation score trends over hourly, six-hour, and daily intervals. Finally, know which monitors are “soft” and which are “hard.” Some monitors should immediately trigger an alert when their threshold is breached. Typically, these are more deterministic monitors evaluating an operational metric such as latency or a system failure. Challenge #3: Flaky Evaluations Who evaluates the evaluators? Using a system that can hallucinate to monitor a system that can hallucinate has obvious drawbacks. The other challenge for creating valid evaluations is that, as every single person who has put an agent into production has bemoaned to me, small changes to the prompt have a large impact on the outcome. This means creating customized evaluations or experimenting with evaluations can be difficult. Best Practices for Avoiding Flaky Evaluations Most teams avoid flaky tests or evaluations by testing extensively in staging on golden datasets with known input-output pairs. This will typically include representative queries that have proved problematic in the past. It is also a common practice to test evaluations in production on a small sample of real-world traces with a human in the loop. Of course, LLM judges will still occasionally hallucinate. Or as one data scientist put it to me, “one in every ten tests spits out absolute garbage.” He will automatically rerun evaluations for low scores to confirm issues. Challenge #4: Visibility Across the Data + AI Lifecycle Of course, once a monitor sends an alert, the immediate next question is always: “Why did that fail?” Getting the answer isn’t easy! Agents are highly complex, interdependent systems. Finding the root cause requires end-to-end visibility across the four components that introduce reliability issues into a data + AI system: data, systems, code, and model. Here are some examples: Data Real-world changes and input drift. For example, if a company enters a new market and there are now more users speaking Spanish than English. This could impact the language the model was trained in.Unavailable context. We recently wrote about an issue where the model was working as intended but the context on the root cause (in this case a list of recent pull requests made on table queries) was missing. System Pipeline or job failuresAny change to what tools are provided to the agent or changes in the tools themselves. Changes to how the agents are orchestrated Code Data transformation issues (changing queries, transformation models)Updates to promptsChanges impacting how the output is formatted Model Platform updates its model versionChanges to which model is used for a specific call Best Practices for Visibility Across the Data + AI Lifecycle It is critical to consolidate telemetry from your data + AI systems into a single source of truth, and many teams are choosing the warehouse or lakehouse as their central platform. This unified view lets teams correlate failures across domains — for example, seeing that a model’s relevancy drop coincided with a schema change in an upstream dataset or an updated model. Deep Dive: Example Architecture The image above shows the technical architecture that Monte Carlo’s Troubleshooting Agent leverages to build a scalable, secure, and decoupled system that connects its existing monolithic platform to its new AI Agent stack. On the AI side, the AI Agent Service runs on Amazon ECS Fargate, which enables containerized microservices to scale automatically without managing underlying infrastructure. Incoming traffic to the AI Agent Service is distributed through a network load balancer (NLB), providing high-performance, low-latency routing across Fargate tasks. The image below is an abstracted interpretation of the Troubleshooting Agent’s workflow, which leverages several specialized sub-agents. These sub-agents investigate different signals to determine the root cause of a data quality incident and report back to the managing agent, who presents the findings to the user. Deliver Production-Ready Agents The core takeaway I hope you walk away with is that when your agents enter production and become integral to business operations, the ability to assess their reliability becomes a necessity. Production-grade agents must be observed. This article was co-written with Michael Segner.

If you’ve built Lightning Web Components (LWC) at scale, you’ve probably hit the same walls I did: duplicated logic, bloated bundles, rerenders that come out of nowhere, and components that were never meant to talk to each other but somehow ended up coupled. When I first transitioned from Aura and Visualforce to LWC, the basics felt easy: reactive properties, lifecycle hooks, and clean templates. But as our team started building enterprise-grade Salesforce apps dozens of screens, hundreds of components the cracks started showing. Performance dipped. Reusability turned into a myth. New devs struggled to onboard without breaking something. This article shares what helped us break that cycle: reusable component patterns, scoped events, smart caching, and render-aware design. Why Reusability and Performance Are Critical in LWC Salesforce isn't just a CRM anymore; it's an app platform. You’re often dealing with: Complex UIs with dynamic layoutsAPI-heavy backends and Apex controller logicStrict governor limitsTeams of developers contributing across multiple sandboxes In this kind of environment, the usual “build fast and clean later” approach doesn’t work. Reusable patterns and performance principles aren’t just nice to have; they’re essential. Especially when each render or Apex trip costs you time, limits, and UX points. Pattern 1: Composition Over Inheritance (and Over Nesting) We started with the common mistake of creating huge parent components that owned every little UI detail dropdowns, modals, tables, loaders. Changes became brittle fast. Instead, we now follow strict composition rules. If a piece of UI can stand on its own (e.g., lookup-picker, pagination-control, inline-toast), it becomes its own component. No logic leaks out. Inputs are exposed via @api, outputs via CustomEvent. Example: HTML CopyEdit <!-- parent.html --> <c-pagination-control current-page={page} total-pages={totalPages} onpagechange={handlePageChange}> </c-pagination-control> This way, our parent component never touches DOM methods or layout tricks. It just delegates and listens. Pattern 2: Stateless Presentational Components We borrowed a page from React and introduced what we call stateless presentational components. These components render only what they’re told: no Apex calls, no wire service, no @track. They just take inputs and return markup. This helped us test faster (no mocking wire/adapters), reuse components in record pages, and reduce side-effects that used to cause reactivity bugs. Pattern 3: Event Contracts and Pub/Sub Boundaries The LWC CustomEvent model is clean until your app grows. We started seeing cascading rerenders because a modal deep in the DOM fired an event that the app shell listened to (via window.dispatchEvent and pubsub). Messy. We introduced event contracts: each component has a known set of events it can emit or consume. No rogue dispatchEvent calls. Pub/Sub boundaries are scoped to app sections, not global. We even versioned events using string names like product:updated:v2. This small process change reduced production event bugs by 40%. Pattern 4: Conditional Rendering vs. DOM Fragmentation If you’re using {#if} in LWC, be careful, it detaches and destroys the entire subtree. We had a dashboard that rerendered every chart from scratch when toggling a filter. CPU spikes, layout shifts, and ugly flickers. The fix? Use hidden or style.display = "none" if you just need to hide, not destroy. Reserve {#if} for full control when data changes significantly. Also, beware of uncontrolled DOM growth. One report page of ours had over 12,000 nodes due to lazy filtering logic and nested <template for:each> inside loops. A quick audit and refactor brought render time down from 1.8s to under 300ms. Pattern 5: Local Storage and Caching Wisely Don’t refetch everything on every load. For components that rely on config data (e.g., picklist values, role maps, branding info), we use a cache strategy: SessionStorage for session-scoped valuesLocalStorage for persistent feature flags or read-only configLightning Data Service for record-backed state We also memoize Apex calls using a keyed map inside @wire or connectedCallback. Result: our homepage boot time dropped by 20%. Pattern 6: Lazy Loading and Dynamic Imports This one's still underused in the LWC world. If your component loads third-party libraries or expensive JS modules (like Chart.js or D3), use loadScript() or dynamic import() to defer until truly needed. JavaScript CopyEdit connectedCallback() { if (!this.chartLoaded) { loadScript(this, CHART_JS) .then(() => this.chartLoaded = true); } } We applied this to our analytics tab and shaved 600KB from the initial bundle. Testing and Linting for Reusability We enforce these rules in CI using: ESLint with LWC pluginJest tests for logic-heavy componentsStorybook for visual regression and documentation Our component PRs require usage examples and at least one story. That change alone made it easier for QA and business analysts to validate features early. Final Thoughts We didn’t arrive at these patterns overnight. Each one came from a specific failure: a broken layout, an unresponsive tab, a hard-to-maintain legacy page. The thing with Salesforce LWC is: it works well for small teams and simple UIs, but as complexity grows, you need rules and patterns that scale with it. By treating components as atomic, stateless, and independently testable and by drawing clear performance boundaries we turned a slow, tangled UI into a platform others could build on. If you’re struggling with slow loads, buggy renders, or a component mess that keeps growing, try some of these patterns. And share your own. We’re all still learning what “scalable” means in Salesforce LWC.

In data storage, the idea of a data lakehouse has transformed data storage and analysis in organizations. Data lakehouses combine the low-cost and scalability storage ability of data lakes and data warehouse’s reliability and performance. In this space, some players have emerged, such as Delta Lake, as strong open-source frameworks for implementing robust ACID-compliant data Lakes. Now, with the introduction of Delta Lake 4.0 and the development of Delta Kernel, the future of the lakehouse architecture is in a revolutionary transition. Brimming with features driving performance, scaling, and interoperability, these updates are to keep up with the increasing dynamics of data workloads in 2025 and beyond. Evolving Data Flexibility: Variant Types and Schema Widening Probably one of the most significant changes in Delta Lake 4.0 is the introduction of the VARIANT data type, which can store semi-structured data without a rigid schema. This is a dramatic change for developers and data engineers who deal with telemetry, clickstream, or JSON-based marketing data. Semi-structured data had to be ‘flattened’ or stored as strings — both of which added complexity and performance limitations. The data can now be stored in raw form as VARIANT, enabling more flexible querying and ingestion pipelines. Interesting to go along with this is type widening, which makes the evolution of table schemata as time passes more straightforward. The field types are usually required to change as data applications grow. For instance, an integer may become a column, but later on, it will need larger values, meaning it has to be transformed into a long type. Delta Lake 4.0 makes such changes with grace, without rewriting entire datasets. Developers can change column types by doing it manually or letting Delta Lake take care of it automatically during inserts and merges, which will decrease the operational overhead and maintain the fidelity of the data historically. Such innovations are an indication of a greater trend in the data world: expanding the needs for systems, which change with the times, not ones that oppose them. Boosting Reliability and Transactions With Coordinated Commits As data transactions are scaled across organizations, transactional consistency across processes and users is of great importance. Delta Lake 4.0 introduces a groundbreaking innovation in this field with Coordinated Commits. This characteristic installs a centralized commit coordination mechanism that ensures multiple users or systems updating the same Delta table are in a synchronized state. Imagine a case where several data pipelines are updating various parts of a table in several clusters at the same time. Inconsistencies and reading anomalies are a danger without coordination. Coordinated Commits make sure that all changes are versioned and separated, which introduces true multi-statement and multi-table transactional capabilities into the lakehouse context. Such a change is essential to organizations that are processing data in a real-time or complex manner of data transformation workflows, where the integrity of data is critical. It sets up Delta Lake’s dream of a very concurrent, multi-user world, and it takes it a step closer to the entire transactional prowess of traditional data warehouses. Remote Interoperability: Delta Connect and Function of Delta Kernel In 2025, there is an increased distribution of data platforms. Data practitioners demand interacting with lakehouses with multiple tools and programming languages, not infrequently remotely and over multiple cloud environments. Delta Lake 4.0 has come with Delta Connect — a feature that is built over Spark Connect that separates the clients’ interface from the data engine. This adds remote access to Delta tables from lightweight clients, which greatly facilitates the connection with notebooks, APIs, and third-party services. Bringing the ability to write an application in Python or JavaScript that can go and read and write directly into Delta tables on remote Spark clusters makes possible what Delta Connect enables. This flexibility enables more nimble development and provides real integration with modern cloud-native tooling. However, what powers the smooth interoperation is the Delta Kernel. Firstly, initially introduced to unify and stabilize the core Delta table protocol, Delta Kernel currently provides a collection of libraries, written in Java and Rust, revealing a clean and consistent interface to Delta tables. These libraries hide internal complexities of partitioning, metadata processing, and the deletion vectors, which makes the adoption of external engines to natively support Delta much simpler. Such projects as Apache Flink and Apache Druid have already implemented Delta Kernel with stunning results. In Flink, with streamlined access to table metadata, Delta Sink pipelines are now in a position to start much faster. In Rust ecosystem, delta-rs have embraced Delta Kernel to allow advanced table operations directly from Python and Rust surroundings. Delta Connect and Delta Kernel combined are making Delta Lake the most available and engine-agnostic lakehouse offering for today. Smarter Performance: Predictive Optimization and Delta Tensor The balancing act of performance management in data lakes has always been the case. Over a period of time, small files, fragmented partitions, and metadata bloat can severely impact performance. Delta Lake tries to overcome this by introducing predictive optimization — a maintenance feature that automatically executes such operations as compaction according to the workload patterns observed. Predictive optimization does not require data engineers to schedule optimize or vacuum commands manually because it tracks the way in which data is queried and maintained. It smartly performs only optimizations as needed, optimizing storage costs, minimize compute usage, maintain high query performance at all times. Such automation is an effort towards self healing data platforms which self-optimize as time passes on like autonomous databases. Another invention promising wide implications is Delta Tensor — a new feature focused on AI and machine learning workloads. While AI adoption is currently soaring high, the need for data scientists to store high-dimensional data, such as vectors and tensors, directly in the lakehouse tables becomes increasingly necessary. Delta Tensor brings support for storing multidimensional arrays in Delta tables with compact, sparse encodings. This is not only a framework for structured and semi-structured data but a viable base for data-rich machine learning systems too. As more machine learning and AI are baked into companies' core products, native support for tensor data in their data platforms is a game-changer. Conclusion Moving through the year 2025, it’s apparent that Delta Lake and its rapidly growing ecosystem have established a new standard for the way data is saved, processed, and operationalized. By integrating data lake scalability with the reliability and performance of the data warehouses, Delta Lake is transforming the landscape of modern data architecture. As the use of Delta Lake 4.0 and Delta Kernel indicates, no matter whether for agile startups or global enterprises, there is a strategic move towards more intelligent, flexible, and interoperable data solutions. With increasing data volumes and changes in analytical needs, these innovations are expected to become key pillars in the future of an enterprise data platform.

The expectations are cosmic. The investments are colossal. Amazon, Google, Meta, and Microsoft collectively spent over $251 billion on infrastructure investment to support AI in 2024, up 62% from 2023's $155 billion, and they plan to spend more than $300 billion in 2025. The prize for those who can provide "superior intelligence on tap," as some are now touting, is infinite. The AI ecosystem is exploding, with new startups and innovative offerings pouring out of global tech hubs. The technology isn’t just evolving; it’s erupting. The theory of AI adoption is also evolving. While everyone acknowledges that risk remains high and vigilance is necessary, concerns are shifting from Terminator-style apocalyptic fantasies to the practical realities of global social disruption anticipated, as AI’s impact cascades through, well, everything. As is becoming clear, we’ll be living next to and collaborating with AI interfaces in every form, from phones and smart glasses to robots and drones. Current academic and industry research strongly supports the thesis that AI-human interaction is evolving toward collaborative teamwork rather than job displacement. The academic community has embraced the term "hybrid intelligence" to describe this phenomenon. Wharton research characterizes hybrid intelligence as "a transformative shift toward a more holistic, human-centered approach to technology and work." The World Economic Forum has introduced "collaborative intelligence" as a framework where "AI teammates will adapt and learn to achieve shared objectives with people." IBM frames this evolution as "an era of human-machine partnership that will define the modern workplace," indicating widespread recognition that superior outcomes emerge from combining human and AI strengths rather than replacing human capabilities. Step back for a second and consider a possible future state of AI integration, in which each team in your organization is partnered with a "superior intelligence in the cloud." Teams will need to be competent at raising the right questions, considering AI responses, evaluating them against our criteria, and reaching consensus between AI and human teammates. Now consider the set of skills required to do this successfully. Preparing AI to collaborate will require training it on data that is pertinent to your business. While the generalized "knowledge on tap" model of public-facing AIs like Claude and Gemini are endlessly useful, specialized models trained on domain- and enterprise-specific data will be able to provide unique insights and combinations not apparent to humans. To collaborate well, AI interfaces will be dynamic, evolving into a more capable partner as it experiences and adapts to your team’s style. To thrive in this evolving environment, organizations must embrace a new paradigm: human-AI readiness. The Indispensable Human-AI Partnership The current theory of AI readiness hypothesizes that machines will not replace knowledge workers, but rather enhance, automate, and rationalize their tasks. In this 'happy path' scenario, AI acts as a powerful augmentative force, applying its research capabilities, its interactive persona, and its ability to absorb, summarize, and interrogate data to aid every human endeavor. AI can serve multiple roles for human collaborators: A thoughtful 'whiteboard,' generating a dialog about possibilities and ideas;An administrative assistant, performing routine administrative tasks, and enabling more time for innovation.An innovation lab, capable of producing prototype ideas, generating specifications or code, performing simulations, or conducting statistical analysis. A constructive critic, reviewing your creative output to ensure clarity, guiding your output to its best presentation. As already indicated by the amount of creative content being produced with AI text, image, and video generators, in the happy-path world, AI becomes a launchpad for human creativity, guided by human intent. Whether AI is being applied in business, the arts, or the military, it has no intent; only humans can supply that. Humans retain control, driving the strategic and creative direction, selection, and refinement, while AI brings breadth of research and analysis capabilities, and the power to generate useful insights. The true power of this transformation, and the competitive advantage it confers, can only be unlocked if all employees are equipped and empowered to use AI effectively. The challenge to this human-AI collaboration model is the widespread variation in AI literacy. Many companies find themselves in the experimentation stage of AI adoption, with limited enterprise-wide proficiency. A majority of workers (78%) want to learn to use AI more effectively, while large segments remain AI avoidant (22%) or merely AI familiar (39%), the category for those informally test-driving some AI tools, but not integrating them into their work yet. Leaders who are inundated with messaging telling them AI is coming to disrupt their business show higher proficiency: 33% AI-literate and 30% AI-fluent. Generational and team-based gaps also exist, and teams like IT and marketing are more AI literate than sales and customer experience (CX). Very few surveyed believed that they were successfully integrating AI into their enterprise in a structured way. A Structured Path to AI Maturity: Organizational Design and Strategic Transition Achieving enterprise-wide AI-enablement and realizing its full potential demands a strategic approach that goes beyond technological implementation. Enterprises that hope to gain market advantage for the strategic application of AI require a roadmap toward AI maturity, in which organizations integrate AI holistically across their enterprise, AI-enabling their data, infrastructure, software stack, model selection and management, as well as the human and change-management disciplines we’ve discussed. Only a small fraction of firms, 12% of companies globally, labeled as AI achievers, have advanced their AI maturity enough to achieve superior growth and business transformation. For these Achievers, AI transformation is an imperative that has driven them to the highest level of urgency and commitment. Achieving AI maturity for these top performers is not defined by any single competency, but by their balanced approach to AI evolution in their enterprise. Accenture's research identifies five key success factors that distinguish AI achievers: Champion AI as a Strategic Priority for the Entire Organization, with Full Sponsorship from Leadership: AI Achievers are significantly more likely to have formal senior sponsorship for their AI strategies, with 83% having CEO and senior sponsorship compared to 56% of experimenters. This executive buy-in is crucial, as strategies without it risk floundering due to competing initiatives.Bold AI strategies, even with modest beginnings, spur innovation and embed a culture of innovation across the organization. Leaders encourage experimentation and learning, implementing systems that help employees showcase innovations and seek feedback.Invest Heavily in Talent to Get More from AI Investments: This is a critical step in bridging the "literacy gap" that holds many companies back from optimizing AI use. AI Achievers prioritize building AI literacy across their workforces, evident in the 78% of them that have mandatory AI training for most employees, from product developers to C-suite executives.This ensures that AI proficiency starts at the top and permeates the organization, making human and AI collaboration scalable. Achievers also proactively develop AI talent strategies/ This systematic re-skilling and talent development is a core component of organizational design for the AI era.Industrialize AI Tools and Teams to Create a Strong AI Core: An AI core is an operational data and AI platform that balances experimentation and execution, allowing firms to productize AI applications and seamlessly integrate AI into other systems. This directly addresses the "technology gap" where businesses lack proper AI communication tools and strategies.Achievers build this core by harnessing internal and external data, ensuring its trustworthiness, and storing it in a single enterprise-grade cloud platform with appropriate usage, monitoring, and security policies. They are also more likely to develop custom machine learning applications or partner with solution providers, tapping into developer networks to swiftly productionize and scale successful pilots. This industrialization ensures that AI isn't siloed but becomes a fundamental part of the business's operational systems.Design AI Responsibly, from the Start: With the increasing deployment of AI, adhering to laws, regulations, and ethical norms is critical for building a sound data and AI foundation. AI achievers prioritize being "responsible by design," proactively integrating ethical frameworks and clear usage policies from the outset.This commitment ensures that AI systems are developed and deployed with good intentions, empower employees, fairly impact customers and society, and engender trust. Organizations that demonstrate high-quality, trustworthy, and "regulation-ready" AI systems gain a significant competitive advantage, attracting and retaining customers while building investor confidence. This is crucial for navigating the "systems gap" by building trust and mitigating risks.Prioritize Long- and Short-Term AI Investments: Achievers understand that the AI investment journey has no finish line and continuously increase their spending on data and AI. They plan to dedicate 34% of their tech budgets to AI development by 2024, up from 14% in 2018.Their investments focus on expanding the scope of AI for maximum impact and "cross-pollinating" solutions across the enterprise. This sustained investment ensures that the organization remains at the cutting edge, continuously improving its AI capabilities and fostering a culture of long-term innovation. These success factors collectively form a comprehensive roadmap for enterprise-wide AI adoption. They involve clear actions such as assigning AI business drivers, educating leadership, engaging employees through interactive sessions, showcasing early wins, launching tailored AI onboarding programs, promoting continuous learning, and creating acceptable usage guidelines and policies. By standardizing tools, training, and processes, businesses can ensure that innovation up-levels all teams, not just a few departments. Redefining Job Duties and Human-AI Interaction As AI becomes deeply embedded in daily workflows, the nature of individual job duties and the very fabric of human-AI interaction will evolve. For technical professionals, understanding the nuances of how users engage with AI-infused systems is paramount for successful implementation and adoption. AI systems, due to their probabilistic nature and continuous learning, can sometimes exhibit unpredictable or inconsistent behaviors, potentially leading to confusion, distrust, or even safety issues. Therefore, designing for effective human-AI interaction is crucial to ensure that people can understand, trust, and effectively engage with AI. By adhering to these guidelines, technical professionals can design and deploy AI solutions that are not only powerful but also user-centric, fostering effective human-AI collaboration in every role. This impacts job duties by shifting focus from manual execution to strategic oversight, creative direction, and problem-solving augmented by AI's capabilities. Conclusion The journey to human-AI readiness is a strategic imperative for every organization. It is a long-term shift that requires proactive planning, incremental adjustments, and a willingness to adapt. The future of business success lies in mastering the "art of AI maturity," integrating cutting-edge technology with thoughtful strategies, robust processes, and, most importantly, an empowered, AI-literate workforce. By championing AI from the top, investing heavily in talent, industrializing AI capabilities, designing responsibly, and making sustained investments, businesses can bridge existing gaps and truly transform their operations. The goal is to create an environment where humans and AI operate as a seamless team, unlocking unprecedented levels of creativity, productivity, and innovation, and ultimately, securing a lasting competitive advantage.

Ensuring your enterprise SaaS application remains always available is more than just a technical objective; it’s a fundamental business requirement. Even short periods of downtime, such as those during routine software updates, can disrupt customers’ operations, erode their trust, and lead to contractual penalties if service-level agreements aren’t met. SaaS applications serve users across multiple time zones. Scheduling downtime that accommodates all users is impractical, making zero-downtime upgrades essential for global businesses. Zero-downtime processes allow for quicker deployment of features, bug fixes, and security patches, supporting agile development and reducing time-to-market. Mastering zero-downtime upgrade procedures, in tandem with robust multi-cloud and multi-region architectures, is essential in maintaining service availability. Recent high-profile outages on major cloud platforms have underscored the disruptive impact of unexpected downtime, affecting organizations worldwide across industries. By distributing workloads across multiple cloud providers and hosting applications in multiple geographic regions, organizations can reduce single points of failure and improve resilience against large-scale outages. If one cloud provider or region experiences issues, traffic can be quickly rerouted to healthy environments, minimizing the impact on end users. This article outlines the key architectural patterns, deployment strategies, and operational best practices for achieving zero-downtime upgrades in Enterprise SaaS environments. Why Zero-Downtime Matters Enterprise customers often depend on SaaS applications for their core business operations. Zero-downtime upgrades ensure that users can access the application without interruption, enhancing productivity and trust. Scheduled maintenance windows are increasingly impractical due to globalization, and unexpected outages harm both the provider’s reputation and customer satisfaction. Ensuring seamless deployments directly supports business continuity and compliance obligations. Also, many enterprise agreements and industry regulations require strict service uptime guarantees. Zero-downtime upgrades help providers consistently meet Service Level Agreements (SLAs) and compliance mandates. Seamless upgrades reduce the risk of errors and complications associated with manual interventions, rollback procedures, or customer complaints due to outages. Foundational Strategies Decouple Deployment and Release Deploy changes safely without immediately exposing them to end users. Incorporate feature flags to gradually enable or disable new features in production environments. One of the most effective ways to deploy new functionality safely is by decoupling deployment from release. This approach ensures that code changes can be pushed to production without instantly impacting end users. The key enabler here is feature flags (also known as feature toggles). Feature flags allow teams to control the visibility and activation of specific features in real time. Instead of waiting for a big-bang release, you can deploy code continuously and then gradually enable features for selected user groups, environments, or regions. This provides several advantages: Progressive delivery: Roll out new functionality to a small percentage of users first, monitor system behavior and user feedback, and expand the rollout only after confirming stability.Instant rollbacks: If a new feature introduces performance regressions or unexpected bugs, it can be disabled instantly — no need to redeploy or revert code.A/B testing and experimentation: Teams can run experiments by enabling or disabling certain features for different cohorts, collecting data, and making data-driven decisions about what to promote to all users.Operational safety: When deploying infrastructure or backend changes, feature flags help teams control risk by activating changes only after confirming all dependent systems are ready. In practice, modern teams often integrate feature flag management directly into their CI/CD pipelines. Tools such as LaunchDarkly, Unleash, or Flagsmith provide dashboards and APIs to manage flag states dynamically. This allows DevOps or SRE teams to toggle features on or off safely without code changes. Ensure Backward Compatibility Design service interfaces and database schemas so that both old and new application versions can operate simultaneously during deployments and rollbacks. Backward compatibility is essential for maintaining system stability during rolling deployments, blue-green releases, and emergency rollbacks. The goal is to ensure that both old and new versions of your application can coexist seamlessly — whether it’s for a few minutes or several days. When designing APIs, service interfaces, or database schemas, follow patterns that allow incremental change rather than breaking change. For example: Version your APIs: Introduce new API versions (/v2/) instead of modifying existing endpoints. This allows older clients to continue operating without disruption while new clients can adopt enhanced functionality at their own pace.Use additive changes: Whenever possible, make schema changes additive. Add new columns or tables instead of renaming or dropping existing ones. Deprecated fields can remain in use until all dependent services are updated.Graceful data migrations: Perform schema migrations in multiple stages—first deploy a backward-compatible change, then update the application logic, and finally remove deprecated elements in a later release cycle.Schema evolution for event streams: For event-driven architectures, ensure your message schemas (e.g., Avro, Protobuf, JSON) can tolerate unknown fields. This allows producers and consumers to evolve independently without breaking the message flow.Contract testing: Implement automated contract tests between services to verify that new versions respect existing integration expectations. Tools like Pact or Spring Cloud Contract help detect breaking changes early in the CI/CD pipeline. Backward compatibility not only supports zero-downtime deployments but also dramatically reduces the risk during rollbacks. If a new release fails, you can revert to the previous version without corrupting data or breaking API consumers. Automate and Orchestrate Leverage CI/CD pipelines and automated testing to coordinate builds, deployments, validations, and rollbacks. Eliminate manual steps that risk human error. By leveraging CI/CD pipelines and intelligent orchestration, teams can deliver code to production faster, with fewer errors and greater confidence. The goal is to build a system where every deployment—from code commit to rollback — is repeatable, traceable, and low-risk. A well-designed CI/CD pipeline doesn’t just build and deploy code — it coordinates every stage of the release process: Build automation: Start with automated build processes that compile, package, and version your application consistently across environments. This ensures reproducibility and eliminates “works on my machine” issues.Automated testing: Integrate comprehensive testing into the pipeline — unit, integration, end-to-end, and performance tests — to validate changes early. Include smoke tests or canary validations post-deployment to catch environment-specific issues.Environment orchestration: Use infrastructure-as-code (IaC) tools like Terraform, Pulumi to manage environments and configurations declaratively. Combine them with container orchestration tools such as Kubernetes for consistent, scalable deployment workflows.Progressive deployment strategies: Incorporate deployment strategies such as blue-green, canary, or rolling updates directly into your CI/CD pipeline. These enable partial rollouts, automatic monitoring, and fast rollback triggers.Rollback automation: Treat rollback as a first-class citizen in your pipeline. Automate reversion steps, ensuring that a single command — or even an automated alert — can trigger a safe recovery if validation checks fail.Continuous verification: Pair your orchestration with real-time monitoring and observability platforms (like Prometheus) to continuously assess system health during and after deployments. Eliminating manual deployment steps not only reduces human error but also enforces consistency and compliance across teams. Every deployment follows the same tested, version-controlled process, which leads to higher confidence and faster recovery when things go wrong. Invest in Observability Instrument systems to capture application metrics, user activity, error rates, and system health. Enable real-time monitoring and alerting to rapidly detect regressions. Monitoring isn’t optional: it’s your early warning system. In addition to standard metrics, invest in: Log analytics: Aggregate and analyze logs in near-real time to spot anomalies.Logging alerts: Configure alerts based on error patterns or service degradations, not just simple thresholds.Dashboards: Build unified dashboards using tools like Grafana to visualize both infrastructure and application performance.Comprehensive metrics: Collect infrastructure, application, and business KPIs to track deployment health and user impact. Deployment Patterns for Zero-Downtime Blue-Green Deployments Maintain parallel production environments (blue and green). Deploy updates to the idle environment, validate stability, and redirect live traffic upon success. Roll back by switching traffic to the previous stable environment if issues arise. Blue-green deployment is a release process that maintains two parallel production environments: typically referred to as “blue” and “green.” At any moment, one environment (say, blue) serves live traffic, while the other (green) remains idle and ready for updates. When it’s time for a new release—whether it’s a bug fix, a major upgrade, or a new feature—the update is deployed to the idle environment. Once validated, the live traffic is switched over, allowing seamless transitions with minimal risk and downtime. This pattern is also called Red-Black Deployment, and naming conventions can vary (“blue” or “green” may represent active or idle environments). The key remains: two identical environments, one live and one ready for change. How Blue-Green Deployment Works The blue-green deployment process consists of four key phases: 1. Setting Up Parallel Environments Start by creating two identical production environments: blue (current live) and green (idle, ready for updates). Having two matched environments ensures that the update can be thoroughly tested without interfering with users. 2. Routing With Load Balancer A load balancer (or router) manages which environment receives user traffic. When it’s time to roll out a new release, the latest code is deployed to the idle environment (green), and all necessary tests and validations are performed. The load balancer then quickly redirects traffic from blue to green, ensuring no DNS propagation issues and a seamless user experience. If issues arise, the load balancer can route traffic back to blue almost instantly. 3. Monitoring the New Release Once the green environment is live and serving production traffic, DevOps engineers monitor its behavior closely. Automated smoke tests and manual health checks ensure the new release’s stability. Any issues or regressions observed can be addressed before they reach a wider audience. 4. Deployment or Rollback If the green environment performs as expected, it officially becomes the primary production environment. The previous blue environment can either be decommissioned or kept as a backup for a short period. If significant issues are identified, a rollback is as simple as switching traffic back to the blue environment, dramatically reducing downtime and user impact. After successful validation and monitoring, the cycle repeats: the now-stable green instance is relabeled as blue, and a fresh environment is prepared for the next release. Recommended for: Complex releases and infrastructure-wide changes.Tradeoff: Increased resource usage during deployment. Canary Releases A canary release, sometimes called a canary deployment, is a proven approach for introducing new changes to production environments with minimal risk. Named after the practice of using canaries in coal mines as early indicators of danger, this technique enables teams to expose new functionality to a limited segment of users or tenants before scaling the update to the entire population. With a canary release, only a small subset — perhaps just 5-10% — of users or servers receive the new code initially. This group acts as the "early warning system." By carefully monitoring error rates, performance metrics, and user experience with this segment, teams can identify any critical issues before they propagate to the broader user base. If metrics remain healthy and no significant errors are detected, the release can be gradually ramped to an increasing percentage of users—eventually encompassing everyone. Conversely, if problems arise, the deployment can be quickly halted or rolled back, preventing the wider user community from being impacted. Crucially, canary releases are built around the following best practices: Segmentation: Release the new version to a clearly defined user or server segment.Automated monitoring: Instrument deployments with monitoring for critical metrics, such as error rates, latency, and resource consumption.Incremental rollout: Gradually increase exposure based on confidence and metric health, rather than releasing to all users at once.Quick rollback: Maintain the ability to rapidly revert to the stable version if anomalies or regressions are detected. This pattern fosters a culture of safety and learning, helping teams mitigate risk and validate real-world impact on a limited scale before committing to a full rollout. By being proactive in tracking stability and user experience, organizations can build confidence in their releases and maintain high availability, even as they innovate and evolve their products. Recommended for: Gradual adoption and risk mitigation in large-scale SaaS environments. Rolling Updates Incrementally apply changes to subsets of servers or containers to maintain continuous service availability. Utilize health checks to verify each batch before progressing. Rolling update, as practiced in orchestrators like Kubernetes, applies changes incrementally to subsets of running pods or containers. For each batch, built-in health checks and readiness probes automatically verify that the new instances are operating as expected before the update proceeds to the next set. If an error or degradation is detected, the update can be paused or reverted to safeguard overall system stability. For example, with Kubernetes, you might start by updating a single pod out of ten. Kubernetes’s controllers monitor this new pod’s health via designated probes — such as HTTP checks or command executions. If the pod becomes "ready," the system updates the next pod, continuing in sequence. This process enables seamless, continuous availability throughout the update, even as new code is introduced. Key Benefits of Incremental Updates Continuous service availability: Only a fraction of servers/containers are unavailable at any moment, reducing the risk of full outages.Automated health verification: Built-in health checks ensure each batch is stable before proceeding.Controlled rollback: If an issue is detected, updates can be halted or reverted, protecting users from widespread impact.Efficiency and speed: Rolling updates balance risk and speed, often allowing frequent, safe updates in fast-moving production environments. By adopting incremental deployment strategies — with robust health checking at each step — you can confidently deliver updates, knowing that issues are contained and service reliability remains uncompromised. Recommended for: Microservice and stateless architectures. Feature Toggles Utilize runtime flags to enable or disable new features dynamically. Leverage runtime feature flags to enable or disable new capabilities instantly, independent of code deployments. This approach offers greater control and flexibility for development teams, allowing them to manage feature exposure dynamically without extensive engineering effort during rollout or testing. Feature flags are invaluable for frequent, incremental product releases and experimentation, as they facilitate A/B testing, rapid rollback of problematic features, and seamless integration of user feedback. Particularly in blue/green deployment strategies, feature flags empower teams to validate new features in a live environment without disrupting existing services — enabling organizations to safely test hypotheses and swiftly disable features that do not meet user expectations. Recommended for: Frequent, incremental product releases and experiments. Schema Changes: Safeguarding Data Availability Managing data migrations without disruption is critical. Use the "expand and contract" pattern: Expand: Introduce new fields or tables in a backwards-compatible manner.Migrate: Update application logic, synchronize, and verify data.Contract: Remove deprecated schema components once the system is fully upgraded. Leverage migration tools (e.g., Liquibase, Flyway) and consider dual-read/write strategies to ensure data consistency. By combining expand and contract principles with robust CI/CD pipelines and observability tooling, teams can make schema changes with zero downtime and minimal user impact. Conclusion Zero-downtime upgrades are an operational standard for modern enterprise SaaS offerings. Success relies on disciplined engineering: resilient architecture, robust automation, comprehensive monitoring, and a customer-centric culture. By applying these patterns and practices, engineering teams can deliver upgrades that delight users — without ever taking the platform offline.

The Edge Observability Security Challenge Deploying an open-source observability solution to distributed retail edge locations creates a fundamental security challenge. With thousands of locations processing sensitive data like payments and customers' personally identifiable information (PII), every telemetry component running on the edge becomes a potential entry point for attackers. Edge environments operate in spaces where there is limited physical security, bandwidth constraints shared with business-critical application traffic, and no technical staff on-site for incident response. Therefore, traditional centralized monitoring security models do not fit in these conditions because they require abundant resources, dedicated security teams, and controlled physical environments. None of them exists on the edge. This article explores how to secure an OpenTelemetry (OTel) based observability framework from the Cloud Native Computing Foundation (CNCF). It covers metrics, distributed tracing, and logging through Fluent Bit and Fluentd. Securing OTel Metrics Mutual Transport Layer Security (TLS) Security of metrics is enabled through mutual TLS (mTLS) authentication, where both client and server end need to prove their identity using certificates before communication can be established. This ensures trusted communication between the systems. Unlike traditional Prometheus deployments that expose unauthenticated HTTP stands for Hypertext Transfer Protocol (HTTP) endpoints for every service, OTel's push model allows us to require mTLS for all connections to the collector (see Figure 1). Figure 1: Multi-stage security through PII removal, mTLS communication, and 95% volume reduction Security configuration, otel-config.yaml YAML receivers: otlp: protocols: grpc: endpoint: mysite.local:55690 tls: cert_file: server.crt key_file: server.key otlp/mtls: protocols: grpc: endpoint: mysite.local:55690 tls: client_ca_file: client.pem cert_file: server.crt key_file: server.key exporters: otlp: endpoint: myserver.local:55690 tls: ca_file: ca.crt cert_file: client.crt key_file: client-tss2.key Multi-Stage PII Removal for Metrics Metrics often end up capturing sensitive data by accident through labels and attributes. A customer identity (ID) in a label, or a credit card number in a database query attribute, can turn compliant metrics into a regulatory violation. The implementation of multi-stage PII removal fixes this problem in depth at the data level. Stage 1: Application-level filtering. The first stage happens at the application level, where developers use OTel Software Development Kit (SDK) instrumentation that hashes out user identifiers with the SHA-256 algorithm before creating metrics. Uniform Resource Locators (URLs) are scanned to remove query parameters like tokens and session IDs before they become span attributes. Stage 2: Collector-level processing. The second stage occurs in the OTel Collector's attribute processor. It implements three patterns: complete deletion for high-risk PII, one-way hashing for identifiers using SHA-256 with a cryptographic salt and use regex to clean up complex data. Stage 3: Backend-level scanning. The third stage provides backend-level scanning where centralized systems perform data loss prevention (DLP) scanning to detect any PII that reached storage, triggering alerts for immediate remediation. When the backend scanner detects PII, it generates an alert indicating the edge filters need updating, creating a feedback loop that continuously improves protection. Aggressive Metric Filtering Security is not only about encryption and authentication, but also about removing unnecessary data. Transmitting less data reduces the attack surface, minimizes exposure windows, and makes anomaly detection easier. There may be hundreds of metrics available out of the box, but filtering and forwarding only the needed metrics reduces up to 95% of metric volume. It saves resources, network bandwidth utilization, and management bottlenecks. Resource Limits as Security Controls The OTel Collector sets strict resource limits that prevent denial-of-service attacks. resourceLimitProtection against Memory 500MB hard cap Out-of-memory attacks Rate limiting 1,000 spans/sec/service Telemetry flooding attacks Connections 100 concurrent streams Connection exhaustion These limits ensure that even when an attack happens, the collector maintains stable operation and continues to collect required telemetry from applications. Distributed Tracing Security Trace Context Propagation Without PII Security for distributed traces can be enabled through the W3C Trace Context standard, which provides secure propagation without exposing sensitive data. The traceparent header can contain only a trace ID and span ID. No business data, user identifiers, or secrets are allowed (see Figure 1). Critical Rule Often Violated Never put PII in baggage. Baggage is transmitted in HTTP headers across every service hop, creating multiple exposure opportunities through network monitoring, log files, and services that accidentally log baggage. Span Attribute Cleaning at Source Span attributes must be cleaned before span creation because they are immutable once created. Common mistakes that expose PII include capturing full URLs with authentication tokens in query parameters, adding database queries containing customer names or account numbers, capturing HTTP headers with cookies or authorization tokens, and logging error messages with sensitive data that users submitted. Implementing filter logic at the application level removes or hashes sensitive data before spans are created. Security-Aware Sampling Strategy Reduction of 90% normal operation traces is supported by the General Data Protection Regulation (GDPR) principle of data minimization while maintaining 100% visibility for security-relevant events. The following sampling approach serves both performance and security by intelligently deciding which traces to keep based on their value. trace typesample raterationale Error spans 100% Potential security incidents require full investigation High-value transactions 100% Fraud detection and compliance requirements Authentication/authorization 100% Security-critical paths need complete visibility Normal operations 10-20% Maintains statistical validity while minimizing data collection Logging Security With Fluent Bit and Fluentd Real-Time PII Masking Application logs are the highest risk involved data, which contain unstructured text that may include anything developers print. Real-time masking of PII data before logs leave the pod represents the most critical security control in the entire observability stack. The scanning and masking happen in microseconds, adding minimal overhead to log processing. If developers accidentally log sensitive data, it's caught before network transmission (see Figure 2). Figure 2: Logging security enabled through two-stage DLP, Real-Time Masking in microseconds, TLS 1.2+ End-to-End, Rate Limiting, and Zero Log-Based PII Leaks Security configuration, fluent-bit.conf YAML pipeline: inputs: - name: http port: 9999 tls: on tls.verify: off tls.cert_file: self_signed.crt tls.key_file: self_signed.key outputs: - name: forward match: '*' host: x.x.x.x port: 24224 tls: on tls.verify: off tls.ca_file: '/etc/certs/fluent.crt' tls.vhost: 'fluent.example.com' Fluentd.conf <transport tls> cert_path /root/cert.crt private_key_path /root/cert.key client_cert_auth true ca_cert_path /root/ca.crt </transport> Secondary DLP Layer Fluentd provides secondary DLP scanning with different regex patterns designed to catch what Fluent Bit missed. This includes private keys, new PII patterns, sensitive data, and context-based detection. Encryption and Authentication for Log Transit Transmission of logs is secured through TLS 1.2 or higher encryption method using mutual authentication. In this communication method, Fluent Bit authenticates to Fluentd using certificates, and Fluentd authenticates to Splunk using tokens. This approach prevents network attacks that could capture logs in transit, man-in-the-middle attacks that could modify logs, and unauthorized log injection. Rate Limiting as Attack Prevention Preventing log flooding avoids both performance and security issues. An attacker generating massive volume of logs can hide malicious activity in noise, consume all disk space causing denial of service, overwhelm centralized log systems, or increase cloud costs until logging is disabled to save money. Rate limiting at 10,000 logs per minute per namespace prevents these attacks. Security Comparison: Three Telemetry Types AspectMetrics (Otel)Traces (Otel)Logs (Fluent bit/fluentd) Primary Risk PII in labels/attributes PII in span attributes/baggage Unstructured text with any PII Authentication mTLS with 30-day cert rotation mTLS for trace export TLS 1.2+ with mutual auth PII Removal 3-stage: App --> Collector --> Backend 2-stage: App --> Backend DLP 3-stage: Fluent Bit --> Fluentd --> Backend Data Minimization 95% volume reduction via filtering 80-90% via smart sampling Rate limiting + filtering Attack Prevention Resource limits (memory, rate, connections) Immutable spans + sampling Rate limiting + buffer encryption Compliance Feature Allowlist-based metric forwarding 100% sampling for security events Real-time regex-based masking Key Control Attribute processor in collector Cleaning before span creation Lua scripts in sidecar Key Outcomes Secured open-source observability across distributed retail edge locationsAchieved Full Payment Card Industry (PCI) Data Security Standard (DSS) and GDPR compliance Reduced bandwidth consumption by 96% Minimized attack surface while maintaining complete visibility Conclusion Securing a Cloud Native Computing Foundation-based observability framework at the retail edge is both achievable and essential. By implementing comprehensive security across OTel metrics, distributed tracing, and Fluent Bit/Fluentd logging, organizations can achieve zero security incidents while maintaining complete visibility across distributed locations.

What if the key to a shared language lay in experience itself? Researchers are now exploring approaches that connect text with images, sounds, and interactions within a three-dimensional world. Sensorimotor grounding, multimodal perception, and world models, all these paths aim to give machines the kind of anchoring they still so painfully lack. Since the machine shares neither our cultural memory nor our perception of the world, several ways can be imagined to bridge that gap. Connecting Language With Real-World Experience The first is to “root symbols in sensorimotor experience.” As early as the 1990s, Stevan Harnad proposed a hybrid model: words should be linked to both iconic representations (images, direct perceptions) and categorical ones (learned invariants), rather than floating in a purely symbolic space. To understand “cat,” then, is not merely to manipulate the word, but above all to connect its use to a perceptual experience, to see it, and ideally one day, to hear and even to touch it. This idea now inspires multimodal approaches, where text and vision are combined to bring linguistic processing closer to grounding in the real world. In practical terms, this amounts to giving the machine a richer form of “experience.” For example, if a model is shown thousands of images of cats accompanied by the caption “cat,” it learns to associate the word not only with other words but also with shapes, colors, and postures. When later asked to describe a photo, it no longer merely manipulates text; it retrieves visual features that refer to a perceptual experience. This combination is what now allows a multimodal model to recognize that “a cat is sleeping on a couch,” instead of merely predicting a string of words unrelated to the image. But here again, the gap between the cognitive abilities of a human and a machine is enormous. As research in vision and cognition reminds us, a young child can recognize a new category with very few examples, sometimes just one, while artificial systems require dozens, hundreds, or even thousands of examples. Teaching Machines to Understand the World In this perspective, spatial intelligence and “world models” play a central role in research. Fei-Fei Li emphasizes the need for AI to reason within a 3D universe, where objects have permanence and physical laws impose constraints. Yann LeCun extends this vision with the concept of “world models”: internal representations that allow systems to simulate, predict, and plan before acting. IBM is part of this same dynamic, working on digital twins for industry and medical research. In concrete terms, these digital twins do not merely represent a “snapshot” of a system, but its “movie.” They make it possible to model both the shape and the evolution of a phenomenon, whether it involves tracking atmospheric currents or understanding how genes interact with one another. All these approaches aim to bring machines closer to the way humans connect language, perception, and action. For my part, I am convinced that these approaches are not mutually exclusive but complementary. None of them will be enough on its own: it is likely by combining embodied perception, efficient processing capabilities, and a solid ethical framework that we will truly be able to move forward. Other lines of research aim instead to adapt operational languages to human intentions. The TransCoder project, for instance, has shown that AI can perform accurate translations between different programming languages (C++, Java, Python) without human supervision. To achieve this, it learned on its own to align its structures and libraries. The level of difficulty is lower compared to understanding human language, since between “machine” languages, meaning is operational and strictly defined. From that starting point, one can hope that it will one day be possible to build an analogous bridge between human language and machine language. The idea would not be to try to imitate our emotions, but to formalize our intentions within an executable protocol. To Be Continued... These approaches outline a future where artificial language would finally be linked to a world of perceptions and actions. But other researchers are choosing a radically different path: rather than imitating human experience, they seek to go beyond its limits by harnessing the power of quantum computing. In the next part, we will dive into the emerging world of Quantum Natural Language Processing. Links to the previous articles published in this series: Series: Toward a Shared Language Between Humans and MachinesSeries (1/4): Toward a Shared Language Between Humans and Machines — Why Machines Still Struggle to Understand Us References Abbaszade, Mina; Zomorodi, Mariam; Salari, Vahid; Kurian, Philip. "Toward Quantum Machine Translation of Syntactically Distinct Languages". [link] Brodsky, Sascha. "World models help AI learn what five-year-olds know about gravity". IBM. [link] Gubelmann, Reto. "Pragmatic Norms Are All You Need – Why The Symbol Grounding Problem Does Not Apply to LLMs". [link]Harnad, Stevan. "The Symbol Grounding Problem". [link]LEO (Linguist Education Online). "Human Intelligence in the Age of AI: How Interpreters and Translators Can Thrive in 2025". [link]Meta AI. "Yann LeCun on a vision to make AI systems learn and reason like animals and humans". [link]Opara, Chidimma. "Distinguishing AI-Generated and Human-Written Text Through Psycholinguistic Analysis". [link]Qi, Zia; Perron, Brian E.; Wang, Miao; Fang, Cao; Chen, Sitao; Victor, Bryan G. "AI and Cultural Context: An Empirical Investigation of Large Language Models' Performance on Chinese Social Work Professional Standards". [link] Roziere, Baptiste; Lachaux, Marie-Anne; Chanussot, Lowik; Lample, Guillaume. "Unsupervised Translation of Programming Languages". [link]Strickland, Eliza. "AI Godmother Fei-Fei Li Has a Vision for Computer Vision". IEEE Spectrum. [link]Trott, Sean. "Humans, LLMs, and the symbol grounding problem (pt. 1)". [link]Nature. “Chip-to-chip photonic quantum teleportation over optical fibers, 2025”. [link]

By Frederic Jacquet CORE

Software development has changed a lot in the past two years. I've been working with AI coding assistants since they first appeared. The most interesting part? It's not just about writing code faster. AI has changed how we validate our products. My co-founder and I noticed something strange on our latest project. Our team was shipping features super fast. But we also had more edge cases and security issues. This is the new reality. You move faster, but things get more complex. Most teams using AI tools face this. The Real Impact Is About Validation, Not Just Speed Let's look at what the numbers tell us. Developers using AI assistants finish coding tasks about 55% faster. That's a big deal, but it's not the whole story. The real win shows up after you write the code. Teams using AI see their time to production drop by 55%. Most of that time savings comes from two things: writing the initial code and getting through the first code review. Why does code review go faster? AI code follows standard patterns more than human code does. It's usually cleaner and more consistent. Even experienced developers writing under pressure don't match this consistency. Reviewers spend less time on style issues and basic mistakes. They can focus on architecture and business logic instead. Accenture tracked its numbers carefully. After adding AI tools, they saw pull requests go up by 8.69%. Merge rates improved by 15%. The best part? Successful builds jumped by 84%. This isn't just faster coding. It's better code getting validated faster. How Developers Are Actually Using These Tools Your role shifts when you use AI tools. You're not really "coding" anymore in the old sense. You're orchestrating. You manage what the AI produces and steer it toward your solution. I spend more time now writing clear instructions and reviewing code than typing implementations. It's like being a tech lead who manages a very fast but unpredictable junior developer. This changes what skills matter. You need to be good at: Breaking problems into clear, small tasksWriting detailed technical specsSpotting when AI code misses contextAsking good questions like "Why did you do it this way?" Some developers struggle with AI tools. They often treat them like magic. The developers who succeed treat them like powerful assistants that need clear direction. The Internal Validation Revolution With Testing at AI Speed I've found test generation to be one of the best uses for AI. Writing unit tests is tedious. Most developers don't enjoy it, even though they know it matters for quality. AI changes this completely. You can create full test suites in minutes instead of hours. But there's a trick to it. You need to guide the AI well. Use advanced prompts like Chain-of-Thought. This means you ask the AI to explain its thinking step by step. It produces much better test coverage. Don't ask: "write tests for this function." Instead ask: "analyze this function, find all edge cases, explain your reasoning, then write full tests for each case." The difference is huge. Simple prompts give basic tests. Structured prompts with reasoning give tests that catch bugs. This automation speeds up internal validation a lot. You're not just building features faster. You're validating them faster, too. External Validation Means Getting to Users Faster The other side of validation is showing your product to real users. AI tools let you go from idea to working prototype in days, not weeks. I recently built an internal dashboard for tracking user metrics. Five years ago, this would have taken a week. With AI help, I had a working prototype in six hours. Not production-ready, but good enough to show stakeholders and get feedback. This speed boost is powerful for testing products. You can test more ideas, fail faster, and iterate based on real feedback instead of guesses. But here's the catch. If you build prototypes 10x faster, you need to collect feedback 10x faster too. Otherwise, you just move the bottleneck. The successful teams build feedback loops into their MVPs from day one. They add analytics, user interviews, and usage metrics. They treat feedback as part of the core product, not something to add later. The Security Problem Nobody Talks About Here's an uncomfortable truth. AI code often has security holes. Not because the AI is malicious. It just lacks context about your security needs. Research shows AI assistants fail at common security tasks: Cross-site scripting protection: 86% failure rateLog injection prevention: 88% failure rateInput sanitization: consistently bad The really dangerous part is psychological. When AI creates code quickly and confidently, you trust it more. You scrutinize it less. This is human nature, but it's also a risk. I've caught myself doing this. The AI creates a database query. It looks fine. I merge it without thinking about SQL injection. That's a problem. The solution is process, not perfection. You need: Static analysis tools (SAST) in your IDEDynamic testing (DAST) for runtime checksSoftware composition analysis (SCA) for dependenciesAutomated security scans in your CI/CD pipeline Some companies use AI security tools trained for finding vulnerabilities. These tools cut detection time by 92% compared to manual reviews. You fight AI problems with AI solutions. The Trust Factor: Why Acceptance Rates Matter Not all AI adoption is equal. Some developers see huge productivity gains. Others barely benefit. The difference is trust. Developers who accept about 30% of AI suggestions report huge benefits. Developers who accept only 23% see minimal gains. That 7% difference matters a lot. Why? Reviewing and rejecting AI suggestions takes time. If you constantly throw away what the AI makes, you waste the time you saved writing code. This is why tracking adoption matters. Measure: Daily active use of AI toolsCode suggestion acceptance ratesLines of AI code that reach production These metrics show whether your team benefits from AI or fights with it. Choosing Your Tools in the Current Landscape The market for AI coding tools has exploded. As of late 2025, over 15 million developers use GitHub Copilot. That's up 400% from last year. But Copilot isn't your only choice. Here's a quick overview of what's available: Full-featured AI coding assistants: GitHub Copilot – Best IDE integration, strong at autocomplete and function generationCursor – Built for AI-first development, excellent chat interfaceAmazon CodeWhisperer – Strong for AWS-specific development AI-powered development platforms: Replit – Browser-based coding with AI assistance built inMimo – Combined learning platform and AI-powered builder for rapid prototypingBolt – Quick full-stack app generation with preview environmentsv0.dev – Specialized for React component generation Autonomous agents (experimental): Devin – Can handle complete features independentlyJules – Focuses on multi-step implementation tasks Most enterprises initially standardize on a single primary tool for cost control and to simplify security reviews. But it's worth budgeting for experimentation. Different tools excel at different tasks. For rapid prototyping and validation, platforms like Mimo or Replit can get you from zero to working prototype faster than traditional IDEs. For production development, GitHub Copilot or Cursor provides better integration with your existing workflow. What This Means for Your Career If you're a developer, you might wonder: "Will AI replace me?" The honest answer is no. But your job is changing. You're moving up the stack. Less time on implementation. More time on architecture. Less time on boilerplate. More time on strategic decisions. Junior developers have an interesting opportunity. AI handles the grunt work that used to take 60% of a junior's time. This means juniors can focus earlier on system design and business logic. Skills that used to take years to develop. Senior developers face a different challenge. Reviewing AI output creates overhead. You context-switch constantly between writing specs, reviewing code, and fixing AI mistakes. The seniors who adapt well embrace the orchestrator role. They get good at prompt engineering. They learn to write clear technical requirements. They develop instincts for when AI code is subtly wrong. Looking Forward to Where This Goes Next We're still early with AI-assisted development. The tools will get better at context. The security issues will mostly be solved. The workflows will mature. But the core shift is permanent. Software development is less about typing code now. It's more about managing a hybrid human-AI process. The competitive edge won't go to teams that code fastest. It will go to teams that integrate AI into their whole validation stack. Testing, security, feedback loops. And who manages the complexity that comes with speed? For product validation, this means: Build feedback collection into your MVP from day oneAutomate your whole testing pyramidEmbed security validation in your workflowMeasure trust and acceptance, not just speed The teams doing this well ship validated products at impossible speeds. The teams doing it poorly ship faster but break more things. The choice is yours. The tools are here. The question is whether you're ready to rethink how you validate your software.

The technology landscape is undergoing a profound transformation. For decades, businesses have relied on traditional web-based software to enhance user experiences and streamline operations. Today, a new wave of innovation is redefining how applications are built, powered by the rise of AI-driven development. However, as leaders adopt AI, a key challenge has emerged: ensuring its quality, trust, and reliability. Unlike traditional systems with clear requirements and predictable outputs, AI introduces complexity and unpredictability, making quality assurance (QA) both more challenging and more critical. Business decision-makers must now rethink their QA strategy and investments to safeguard reputation, reduce risk, and unlock the full potential of intelligent solutions. If your organization is investing in AI capabilities, understanding this quality challenge isn’t just a technical concern; it’s a business necessity that could determine the success or failure of your AI initiatives. In this blog, we’ll explore how AI-driven development is reshaping QA — and what organizations can do to ensure quality keeps pace with innovation. Why Traditional Testing Falls Short Let’s take a practical example. Imagine an interview agent built on top of a large language model (LLM) using the OpenAI API. Its job is to screen candidates, ask context-relevant questions, and summarize responses. Sounds powerful, but here’s where traditional testing challenges emerge: Non-Deterministic Outputs Unlike a rules-based form, the AI agent might phrase the same question differently each time. This variability makes it impossible to write a single “pass/fail” test script. Dynamic Learning Models Updating the model or fine-tuning with new data can change behavior overnight. Yesterday’s green test might fail today. Contextual Accuracy An answer can be grammatically correct yet factually misleading. Testing must consider not just whether the system responds, but whether it responds appropriately. Ethical and Compliance Risks AI systems can accidentally produce biased or non-compliant outputs. Testing must expand beyond functionality to include fairness, transparency, and safety. Clearly, a new approach is needed. AI-Powered Testing So, what does a modern approach to testing look like? We call it the AI-powered test, a fresh approach that redefines quality assurance for intelligent systems. Instead of force-fitting traditional, deterministic testing methods onto non-deterministic AI models, businesses need a flexible, risk-aware, and AI-assisted framework. At its core, AI-powered testing means: Testing at the behavioral level, not just the functional level.Shifting the question from “Does it work?” to “Does it work responsibly, consistently, and at scale?”Using AI itself as a tool to enhance QA, not just as a subject to be tested. This approach ensures that organizations not only validate whether AI applications function, but also whether they are reliable, ethical, and aligned with business goals. Pillars of AI-Powered Testing To make this shift practical, we recommend you plan your AI QA strategy around the following key pillars: 1. Scenario-Based Validation Instead of expecting identical outputs, testers validate whether responses are acceptable across a wide range of real-world scenarios. For example, does the Interview Agent always ask contextually relevant questions, regardless of candidate background or job description? 2. AI Evaluation Through Flexibility AI systems should be judged on quality ranges rather than rigid outputs. Think of it as setting “guardrails” instead of a single endpoint. Does the AI stay within acceptable tone, accuracy, and intent even if the exact wording varies? 3. Continuous Monitoring and Drift Detection Since AI models evolve, testing can’t be a one-time activity. Organizations must invest in continuous monitoring to detect shifts in accuracy, fairness, or compliance. Just as cybersecurity requires constant vigilance, so too does AI assurance. 4. Human Judgment Automation is powerful, but human judgment remains essential. QA teams should include domain experts who can review edge cases and make subjective assessments that machines can’t. For business leaders, this means budgeting not only for automation tools but also for skilled oversight. The Future Is Moving From ‘AI for Testing’ to ‘Testing for AI’ AI is reshaping every part of the technology ecosystem, and software testing is no exception. We have AI-driven test automation tools like robonito, KaneAI, testRigor, testim, loadmill and Applitools. These are powerful allies that use AI to make traditional testing faster and more efficient. They can write test scripts from plain English, self-heal when the user interface changes, and intelligently identify visual bugs. These tools are excellent for improving the efficiency of testing traditional applications. But the real frontier is “AI platforms designed to test other AIs.” This is where the future lies. Think of these as “AI test agents,” specialized AI systems built to audit, challenge, and validate other AI. This emerging space is transforming how we think about quality assurance in the age of intelligent systems. Key Directions in “Testing for AI” LLM evaluation platforms: New platforms are being developed to rigorously test applications powered by LLMs. For example , an Interview Agent can generate thousands of diverse, adversarial prompts to check for robustness, test for toxic or biased outputs, and compare the model’s responses against a predefined knowledge base to check for factual accuracy/hallucinations.Model monitoring and bias detection tools: Companies like Fiddler AI and Arize AI provide platforms that monitor your AI in production. They act as a continuous QA system, flagging data drift (when real-world data starts to look different from training data) and detecting in real-time if the model’s outputs are becoming biased or skewed.Testing for AI: There are many companies working on AI agent testing and agent-to-agent testing tools. For example, LambdaTest recently launched a beta version of its agent-to-agent testing platform — a unified environment for testing AI agents, including chatbots and voice assistants, across real-world scenarios to ensure accuracy, reliability, efficiency, and performance. Why This Matters for Business Leaders From a C-suite perspective, investing in AI-powered testing isn’t just a technical decision; it’s a business imperative. Here’s why: Customer trust. A chatbot that provides incorrect medical advice or a hiring tool that shows bias can damage brand reputation overnight. Quality isn’t just about uptime anymore; it’s about ethical, reliable experiences.Regulators are watching. AI regulation is tightening worldwide. Whether it’s GDPR, the EU AI Act, or emerging US frameworks, organizations will be held accountable for how their AI behaves. Testing for compliance should be part of your risk management strategy.Cost of failure. With AI embedded in core business processes, errors don’t just affect a single user; they can cascade across markets and stakeholders. Proactive QA is far cheaper than reactive damage control.Competitive advantage. Companies that can assure reliable, responsible AI will differentiate themselves. Just as “secure by design” became a competitive market in software, “trustworthy AI” will become a business differentiator. Building Your AI QA Roadmap So, how should an executive get started? Here’s a phased approach we recommend to clients: Phase 1: Assess Current Gaps Map where AI is currently embedded in your systems. Identify areas where quality risks could impact customers, compliance, or brand reputation. Phase 2: Redefine QA Metrics Move beyond pass/fail. Introduce new metrics such as accuracy ranges, bias detection, explainability scores, and response relevance. Phase 3: Invest in AI-Powered Tools Adopt platforms that can automate scenario generation, inconsistency detection, and continuous monitoring. Look for solutions that scale with your AI adoption. Phase 4: Build Cross-Functional Oversight Build a governance model that includes compliance, legal, and business leaders alongside IT. Quality must reflect business priorities, not just technical checklists. Phase 5: Establish Continuous Governance Treat AI QA as an ongoing discipline, not a project phase. Regularly review model performance, monitor for drift, and update guardrails as the business evolves. Final Thoughts The era of AI-driven applications is here, and it’s accelerating. But with innovation comes responsibility. Traditional QA approaches built for deterministic systems are no longer sufficient. By adopting an AI-powered testing strategy, organizations can ensure their AI systems are not only functional but also ethical, reliable, and aligned with business goals. The message for leaders is clear: if you want to harness AI as a competitive advantage, you must also invest in the processes that make it trustworthy. Modern QA is no longer just about preventing bugs; it’s about protecting your brand, your customers, and securing your organization’s future in an AI-first world.

By Kailash Pathak CORE

After years of managing cloud services in a traditional setting — manually provisioning clusters, setting up networks, managing credentials, and navigating deployment scripts — I thought I had mastered the rhythm of delivery. Dashboards, support tickets, and carefully planned change windows were the most important things in my life. It was safe, predictable, and well-organized, but it was also slow, tiring, and full of dependencies that only worked after a lot of planning and careful changes. Then came the shift. Our organization launched a developer platform, which was a single, unified space that promised automation, golden paths, and built-in safety nets. It wasn't just a new tool; it was a whole new way of thinking. I was used to having full control over everything, like manually provisioning, configuring, and deploying. The idea of abstracted infrastructure and self-service pipelines felt new to me. It was both exciting and scary, like going from steering a ship by hand to trusting a smart navigation system to find the way. At first, I hesitated. Could a system this well-organized really handle all the complicated things I had been doing by hand for years? But as I started to learn more about platform engineering by making a workspace, deploying a service, and watching everything connect without any problems, I began to get it. It wasn't about taking power away from developers; it was about getting us out of the never-ending cycle of setup and repetition so we could focus on building what really mattered. This blog is about that transformation: going from a world of hand-made configurations to one powered by platforms, where speed, consistency, and safety finally work together. It's about letting go of old habits, accepting new rules, and understanding that joining a platform doesn't mean giving up control or knowledge; it means gaining more. The Builders and the Bridge For years, developers and operations teams worked towards the same goal, but from different sides: speed versus stability. Developers made things quickly, while ops made sure they worked. The end result was a bigger gap that made progress slower for both. Then came the platform engineers, who built the bridges. They didn't just make another tool; they laid the groundwork for a new way of working. A system that made the boring tasks automatic, made the hard tasks easier, and gave developers the confidence to move quickly without breaking things. That base grew into the internal developer platform (IDP), which finally linked speed and stability. The IDP was no longer just a bridge; it was the road that went across the river. Developers didn't have to wait in queue or raise tickets for every little change to a deployment or configuration anymore. They could go from idea to deployment in a matter of minutes with self-service golden paths. Platform engineers worked behind the scenes to keep the bridge strong and safe. They improved automation, tightened guardrails, and made sure that every release went smoothly from code to production. The result? Innovation didn't just go faster; it also went smarter. In essence, Platform engineering is the craft — the discipline of designing the systems, automation, and guardrails that empower teams.IDP is the product — the tangible outcome of that craft, offering developers a self-service, secure, and consistent way to bring ideas to life. Design Principles of the Platform An effective IDP is more than a collection of tools; it's a carefully thought-out system that strikes a balance between automation, security, and reliability. Its design principles set the rules for how developers work with infrastructure while making sure that things stay the same and are governed on a large scale. Our platform was built on three main pillars: Runtime – The platform should make sure that applications run reliable, scalable, and work the same way in all environments. It makes infrastructure less complicated by automating it, which lets teams deploy, scale, and monitor workloads more easily.Compliance – Governance must be built into the platform. Compliance ensures that every deployment adheres to organizational and regulatory standards through policy enforcement, access control, and audit trails.SRE (Site Reliability Engineering) – Reliability is a shared responsibility. Embedding SRE practices into the platform makes it more resilient by using observability, error budgets, proactive monitoring, and automated recovery. This makes operational excellence a process that can be repeated. The Human Side of Platform Engineering No onboarding experience is perfect. There are still moments of confusion — “Which environment variable maps to which secret?” or “How do I roll back a feature?” But when the platform team responds through community office hours, in-house chatbots, in-portal feedback, or documentation updates, trust grows. A great platform isn’t just a product — it’s a relationship between builders and enablers. “When an issue was encountered, it was fixed by updating the template/automation to fix this for everyone/future. That’s when I realized what ‘platform engineering’ truly means.” My End-to-End Journey of Onboarding to the Platform Moments That Changed How I Work Week 1 The Beginning: A New Developer, A New Platform. When I first joined the team, the word “platform” floated around in every conversation. It was described as the foundation that powered all deployments, managed compliance, and simplified delivery. I didn't start my onboarding process with paperwork; I started it with curiosity.The First Login: Where the Story Begins. I log into the internal developer portal. Portal greets not just by a wall of documentation but by a dashboard of “golden paths.” A golden path is a carefully chosen, opinionated workflow.The Setup Phase: The platform automatically sets up a namespace in Kubernetes, sets resource quotas, and enforces security policies using pre-made templates instead of having to do it by hand. The Golden Path Moment: From Zero to Deployed – I pick a template, fill-in a few details — repo name, image path, environment — and the platform auto-provisions the infrastructure as per the developer requirement.Guardrails, Not Roadblocks: As I navigated deeper, I realized how carefully the platform balanced freedom and governance. There were safeguardrails around every action, from making a namespace to deploying an app.Week 2 The Hidden Power: Shared Responsibility Without Friction: The platform's design includes shared responsibility, security is built in, performance patterns are pre-tested, and cost transparency is clear. There is a clear layer of responsibility for every action a developer takes.Within a Month From Developer to Contributor: The platform’s contribution model allowed me (developer) to extend templates, publish new workflows, and share best practices with peers. This is the final stage of onboarding — when developers become contributors. It’s about co-creating the ecosystem.Closing Thoughts: Onboarding as a Continuous Experience – Platform is a developer experience system. Every problem, every automation, and every feedback loop affects how developers think about productivity and new ideas. When onboarding goes smoothly, developers stop worrying about the platform and start thinking about what they can do with it.The Realization: Freedom with Limits – Initially, I worried that such automation might take away flexibility — the ability to tweak configurations or optimize deployments. But what I found was the opposite: freedom within a framework. The platform took care of the infrastructure scaffolding, compliance, and performance best practices that I used to have to enforce myself, so I could focus on my service logic. Lessons Learned My journey through platform onboarding taught me a few lasting lessons: Simplicity is the greatest enabler – Developers don’t want more tools; they want fewer decisions that matter.Golden Paths reduce cognitive load – Pre-defined workflows and templates (golden paths) guide developers through the right way to build and deploy, minimizing decision fatigue, setup errors, and inconsistency.Guardrails inspire confidence – When governance feels like guidance, compliance stops being a burden.Automation turns reliability into routine – Automated CI/CD pipelines, environment provisioning, and observability integrations replace manual steps — making reliability a default, not an afterthought.Observability is built-in, not bolted on – From metrics to traces, platform onboarding ensures that every service is observable by design, enabling faster debugging, root cause analysis, and proactive performance tuning.Onboarding never truly ends – Great platforms evolve as developers grow, keeping the experience fresh and relevant.A great platform grows with its developers – Platforms thrive when developers feel heard — it turns users into contributors.Culture completes the code – Behind every great platform is a team that listens, iterates, and learns from developer feedback. The human element — empathy, communication, and shared ownership — turns a system into a culture.Collaboration through standardization – When every team followed the same golden paths and governance patterns, it became easy to understand each other’s deployment pipelines, troubleshoot across teams without deciphering custom scripts, and compare performance and cost metrics in a meaningful way.Automation works best when it’s transparent – The platform didn’t hide complexity; it revealed it at the right moments, making me smarter, not passive. The Role of AI in Modern Developer Platforms As platforms mature, AI is becoming a silent teammate in the developer experience. It’s no longer just about automating pipelines; it’s about augmenting intelligence across the entire onboarding journey. While my platform team has identified many futuristic AI solutions to manage the platform, following are the current developer focused assistants, that focuses on enhancing the onboarding journey. Intelligent onboarding assistance – AI-powered assistants embedded in the developer portal guide newcomers contextually — explaining errors, suggesting templates, or even generating YAML manifests on the fly. This results in reduced onboarding time and fewer blockers."re-inviteme" bot – AI-powered chat assistant that helps developers regain or request the right infrastructure access. Developer can run this to create/update their access, which at the background follows the organization policies to provide access to the corresponding developer. "create-config" bot – Chat assistant that helps developers create new golden path configuration."update-config" bot – Chat assistant that helps developers update their existing golden path configuration. AI doesn’t replace the platform engineer — it amplifies them. Closing Thoughts The journey of onboarding to a platform isn’t just about learning how to use the tools or processes to be followed; it’s about experiencing enablement. When a platform makes things easier, it empowers developers to focus on what they love most — building, experimenting, and innovating. A great platform doesn’t just get developers started; it gives them confidence. And that's what I’ve learned: a platform transforms a system into a community.

By Josephine Eskaline Joyce CORE